- 1812 Views
- 1 replies
- 5 kudos
accessing secret from spark cluster.
passing spark configuration to access blob, adls from data factory while creating job clusterit's working fine, but when in the property we are accessing secret it's not workingspark.hadoop.fs.azure.account.auth.type.{{secrets/scope/key}}.dfs.core.wi...
- 1812 Views
- 1 replies
- 5 kudos
- 5 kudos
check here : https://docs.databricks.com/security/secrets/secrets.html
- 5 kudos
- 2550 Views
- 2 replies
- 0 kudos
adding Widget as a column and populating its value every-time in that column in a table.
hi , I want date for runtime from ADF as @utcnow() -- base paramater of notebook activity in ADF and take the data in ADB using widgets as runtime_date, further i want that column to be added in my table X with the populated value from the widget.Eve...
- 2550 Views
- 2 replies
- 0 kudos
- 0 kudos
you can use as current_timestamp() or now()refer link: https://docs.databricks.com/sql/language-manual/functions/current_timestamp.html
- 0 kudos
- 32656 Views
- 6 replies
- 7 kudos
Resolved! What does "Determining location of DBIO file fragments..." mean, and how do I speed it up?
Determining location of DBIO file fragments. This operation can take some time.What does this mean, and how do I prevent it from having to perform this apparently-expensive operation every time? This happens even when all the underlying tables are De...
- 32656 Views
- 6 replies
- 7 kudos
- 7 kudos
Determining location of DBIO file fragments" is a message that may be displayed during the boot process of a computer running the NetApp Data ONTAP operating system. This message indicates that the system is currently in the process of identifying an...
- 7 kudos
- 6654 Views
- 4 replies
- 6 kudos
How do I know how much of a query/job used Photon?
I'm trying to use the native execution engine, Photon. How can I tell if a query is using Photon or is falling back to the non-native Spark engine?
- 6654 Views
- 4 replies
- 6 kudos
- 6 kudos
Typo error in my second point of the previous post. Click the execution plan of your task[this is available under SQL/Dataframe tab in Spark UI]. It explains what operations run in the photon engine and what didn't execute by photon.
- 6 kudos
- 11833 Views
- 9 replies
- 2 kudos
text datatype not supported and data having huge data in text filed how to bring it over
Hello all,I have medical field data file and one of the field is the text field with huge data not the big problem is databrick does not support text data type so how can i bring the data over. i tried conversion, cast in various way but so far not ...
- 11833 Views
- 9 replies
- 2 kudos
- 2 kudos
Setting escapeQuotes to false has helped to bring huge text data in colomn.thanks
- 2 kudos
- 4336 Views
- 3 replies
- 1 kudos
Resolved! How to start my cluster
​I try to stop my_cluster from compute from admin role. BTW, using same account, I could not restart my_cluster. The information is as followings. How should I do?
- 4336 Views
- 3 replies
- 1 kudos
- 1 kudos
rit seems this is community edition so in CE this feature is disabled , delete this one and create new cluster
- 1 kudos
- 2072 Views
- 1 replies
- 2 kudos
Documentation Update January 13 - 19 Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, a...
Documentation Update January 13 - 19Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, and data engineers working in the Databricks Data Science & Engineering, Databricks Machine Learning, ...
- 2072 Views
- 1 replies
- 2 kudos
- 2 kudos
thanks for the details
- 2 kudos
- 3292 Views
- 1 replies
- 0 kudos
Resolved! Import course material to databricks
I signed up for the data engineering course and downloaded the course material.However I cannot access the link to import the course material into databricks. Below link gives me access denied.https://www.databricks.training/step-by-step/importing-co...
- 3292 Views
- 1 replies
- 0 kudos
- 0 kudos
https://github.com/databricks-academy/data-engineering-with-databricks-english use this link and download this to your local and then import, it will work
- 0 kudos
- 12484 Views
- 1 replies
- 0 kudos
Resolved! configuring the Databricks JobAPIs and I get Error 403 User not authorized.
I’m configuring the Databricks JobAPIs and I get Error 403 User not authorized.I found out the issue is that I need to apply a rule and set API permissions for AzureDatabricksAzure Portal>Azure Databricks>Azure Databricks Service>Access control (IAM...
- 12484 Views
- 1 replies
- 0 kudos
- 0 kudos
for the particular jobs the user who is trying to start the job he should have access permission or run permission for that jobs , please give required permission and it will work for sure
- 0 kudos
- 4747 Views
- 3 replies
- 0 kudos
preAction in databricks while writing into Google Big Query Table?
I am writing into Google Big Query table using append mode. I need to delete current day data before writing new data. I just want to know if there is any preActions parameter can be used to first delete data before writing into table? Below is the s...
- 4747 Views
- 3 replies
- 0 kudos
- 1470 Views
- 0 replies
- 0 kudos
Exposing prometheus metrics from inference server in azure machine learning
At the moment we are running an inference server in azure machine learning. We would like to be able to expose existing metrics to prometheus as well as create our own custom metrics, all described belowExpose existing metrics:I would like a breakdow...
- 1470 Views
- 0 replies
- 0 kudos
- 7734 Views
- 2 replies
- 0 kudos
How to add Current date as one of the column in Databricks
I am trying to create new column "Ingest_date" in table which should contain current date. I am getting error "Current date cannot be used in a generated column". Can you please review and suggest alternative to get the current date in delta table.
- 7734 Views
- 2 replies
- 0 kudos
- 0 kudos
A generation expression can use any SQL functions in Spark that always return the same result when given the same argument valuesSource: https://docs.delta.io/latest/delta-batch.html#use-generated-columnsIt means that it's intended to not work.You ca...
- 0 kudos
- 7773 Views
- 3 replies
- 3 kudos
Multiple streaming sources to the same delta table
Is it possible to have two streaming sources doing Merge into the same delta table with each source setting a different set of fields?We are trying to create a single table which will be used by the service layer for queries. The table can be populat...
- 7773 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Zachary Higgins​ Thanks for the replyCurrently, we are also using Trigger.once so that we can handle the merge stream dependencies properly. But was wondering whether we can scale our pipeline to be streaming by changing the Trigger duration in t...
- 3 kudos
- 11713 Views
- 9 replies
- 9 kudos
copy files from azure to s3
I am trying to copy files from azure to s3. I've created a solution by comparing file lists and copy manually to a temp file and upload. However, I just found AutoLoader and I would like to use that https://docs.databricks.com/ingestion/auto-loader/i...
- 11713 Views
- 9 replies
- 9 kudos
- 9 kudos
Just use tools like Goodsync and Gs Richcopy 360 to copy directly from blob to S3, I think you will never face problems like that ​
- 9 kudos
- 3137 Views
- 1 replies
- 6 kudos
Resolved! Logic execute when we create an View?
Hi all,I have a small curious about VIEW on Databricks. Could anyone please help me clarify this?Normal database like Postgres or MS SQL, when we define a view, the logic still not execute that time, only run when we query that VIEW.Not sure how VIEW...
- 3137 Views
- 1 replies
- 6 kudos
- 6 kudos
@Nhan Nguyen​ It works the same. CREATE VIEW constructs a virtual table that has no physical data.
- 6 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
