- 7291 Views
- 1 replies
- 2 kudos
Resolved! Building a Document Store on Databricks
Hello,I am somewhat new to Databricks and am trying to build a Q&A application based on a collection of documents. I need to move .pdf and .docx files from my local machine to storage in Databricks and eventually a document store. My questions are:Wh...
- 7291 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi all,I took an initial stab at task one with some success using the Databricks CLI. Here are the steps below:Open Command/Anaconda prompt and enter: pip install databricks-cliGo to your Databricks console and under settings find "User Settings" and...
- 2 kudos
- 3268 Views
- 3 replies
- 1 kudos
Issues to load from ADLS in DLT
I am using DLT to load csv in ADLS, below is my sql query in notebook:CREATE OR REFRESH STREAMING LIVE TABLE test_account_raw AS SELECT * FROM cloud_files( "abfss://my_container@my_storageaccount.dfs.core.windows.net/test_csv/", "csv", map("h...
- 3268 Views
- 3 replies
- 1 kudos
- 1 kudos
thank you every one, the problem is resolved, problem is gone when I have workspace admin access.
- 1 kudos
- 12630 Views
- 2 replies
- 2 kudos
Facing issue while using widget values in sql script
I am using below code to create and read widgets. I am assigning default value.dbutils.widgets.text("pname", "default","parameter_name")pname=dbutils.widgets.get("pname")I am using this widget parameter in some sql scripts. one example is given below...
- 12630 Views
- 2 replies
- 2 kudos
- 3922 Views
- 0 replies
- 0 kudos
Trying to convert STRING column into Array of Structs in SQL statement
I have STRING column in a DLT table that was loaded using SQL Autoloader via a JSON file. When I use the "schema_of_json" function in a SQL statement passing in the literal string from the STRING column then I get this output:ARRAY<STRUCT<firstFetchD...
- 3922 Views
- 0 replies
- 0 kudos
- 2638 Views
- 2 replies
- 0 kudos
Databricks extension in Visual studio Code
Databricks recently introduced extension inside the VS code, this is good feature but my company has some security concerns, If I wanted to block connecting to Databricks from Visual studio code, How can I do it? Is there any process which blocks con...
- 2638 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Kiran Gogula We haven't heard from you since the last response from @Debayan Mukherjee , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be helpful t...
- 0 kudos
- 6366 Views
- 2 replies
- 1 kudos
Overwriting schema in Delta Live Tables
Hi allI have a table created by DLT. Initially I specified cloudFiles.inferColumnTypes to false and all columns are stored as strings. However, I now want to use cloudFiles.inferColumnTypes=true. I dropped the table and re-ran the pipeline, which fai...
- 6366 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Billy Wong Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 1 kudos
- 6232 Views
- 1 replies
- 1 kudos
Arrays of complex type always evaluate to ARRAY<STRING>?
Arrays of complex types seemingly always evaluate to ARRAY<STRING>. Therefore, casting or attempting to load JSON data with empty array values fails. For example, attempting to cast a JSON value of {"likes": []...} on load to the following table sche...
- 6232 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Jake Neyer Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers yo...
- 1 kudos
- 5469 Views
- 3 replies
- 5 kudos
Resolved! Training models on big or small clusters
I have a workflow with a model which trains every sunday in Azure Databricks. Sometimes the workflow fails as the max wait time is exceeded (currently I am using 1200 seconds). To solve the problem I was thinking of either increasing the wait time or...
- 5469 Views
- 3 replies
- 5 kudos
- 5 kudos
Hi @Andreas Kaae Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 5 kudos
- 7587 Views
- 2 replies
- 4 kudos
Resolved! Workspace not getting created
Hey guys,I tried to create a workspace, I didn't encountered error like this. I provided everything correctly but don't know why I'm getting this. Tried doing it by using different Data bricks Id and AWS ID also access this on AWS Root account
- 7587 Views
- 2 replies
- 4 kudos
- 4 kudos
https://docs.gcp.databricks.com/administration-guide/workspace/create-workspace.html
- 4 kudos
- 2155 Views
- 1 replies
- 1 kudos
Access DBU used per cluster using within Databricks Clusters
Could it be possible, we can retrieve the DBU's on cluster basis within Databricks Notebook itself?This info we get in the compute tab in Databricks for each cluster as Active DBU/hr.
- 2155 Views
- 1 replies
- 1 kudos
- 1 kudos
It wont be possible to access DBU used per cluster within DB Clusters.
- 1 kudos
- 1489 Views
- 0 replies
- 0 kudos
Pulling query history only for the last 5 mins using "/api/2.0/sql/history/queries" api
I know query history api provides filter_by option with start and end time in ms but I was wondering if I can get only the last 5 mins of query data every time I run the api call (using telegraf to call the api). Is it possible I can use relative dat...
- 1489 Views
- 0 replies
- 0 kudos
- 8158 Views
- 0 replies
- 0 kudos
how to read columns dynamically using pyspark
I have a table called MetaData and what columns are needed in the select are stored in MetaData.columnsI would like to read columns dynamically from MetaData.columns and create a view based on that.csv_values = "col1, col2, col3, col4"df = spark.crea...
- 8158 Views
- 0 replies
- 0 kudos
- 12692 Views
- 2 replies
- 1 kudos
Resolved! How to Pull Cached SQL Table into Python Dictionary?
Hello,I have been working on this issue as a proof of concept - it would be extremely helpful to iterate through tables via loops in a few scenarios. I have a simple three column dimension that I added to a cached table.cache lazy table hedis_cache s...
- 12692 Views
- 2 replies
- 1 kudos
- 1 kudos
Got it to work, thank you for the tip! I needed to convert the dataframe over to a pandas dataframehttps://www.geeksforgeeks.org/convert-pyspark-dataframe-to-dictionary-in-python/
- 1 kudos
- 4575 Views
- 4 replies
- 3 kudos
Unity Catalog Primary key column taking duplicates
I have Updated a Hive Meta Store from a Unity Catalog. I have setup Primary keys on the table. When I try to insert duplicates its succeeding Inserts and seems like PK is not working. Anyone else seeing such behaviour ?
- 4575 Views
- 4 replies
- 3 kudos
- 1227 Views
- 0 replies
- 0 kudos
Databricks CI/CD process is decoupled or coupled ?
In Databricks, CI/CD process is decoupled or coupled ?
- 1227 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 579 | |
| 349 | |
| 287 |
