- 1114 Views
- 3 replies
- 0 kudos
How do Databricks handle the utilization of Azure Reserved Instances?
We are trying to use Azure Reserved Instances for Databricks clusters as we have some consistent workload.Does Databricks automatically select the reserved instances when they are available (assuming we keep the on-demand instance type in the cluster...
- 1114 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @NathanL, Does Databricks automatically select the reserved instances when they are available (assuming we keep the on-demand instance type in the cluster config)? Or is there explicit configuration we need to set to have Databricks use those re...
- 0 kudos
- 4451 Views
- 4 replies
- 0 kudos
Insufficient privileges:User does not have permission SELECT on any file
Hello,after switching to "shared cluster" usage a python job is failing with error message: Py4JJavaError: An error occurred while calling o877.load. : org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User...
- 4451 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @sravs_227 ,the issue was, that the checkpoint directory (while reading from kafka) was set to a dbfs folder. We switched this now to also UC volume
- 0 kudos
- 423 Views
- 1 replies
- 1 kudos
Resolved! register for Databricks Community Edition
I've been trying to register for Databricks Community Edition for 3 days, to study the platform, but I'm experiencing an error when registering ("An error has occurred. Please try again later."). Could anyone help me, how do I register for the commun...
- 423 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @JoaoJr, Thank you for contacting Databricks Community Discussion Forum. Please note that for any issues related to the Databricks Community Edition product, you can find helpful resources here. If you encounter any difficulties beyond what's co...
- 1 kudos
- 1410 Views
- 3 replies
- 0 kudos
Feature Store UI lagging
The feature store UI is lagging (Azure Databricks). It is extremely slow to update - takes several days from creating a feature table before it appears in the UI. Anyone share the same experience?
- 1410 Views
- 3 replies
- 0 kudos
- 0 kudos
Anyone having the same experience here? The UI is still extremely slow to update - it takes days before the already registered feature tables appear in the UI. It makes the UI useless for browsing features! I really hope that Databricks is looking in...
- 0 kudos
- 868 Views
- 2 replies
- 1 kudos
Maintaining Order Consistency: Table Creation in Databricks SQL vs. DLT Pipeline
I have a CTE table with the below names as values. My objective is to create another table by concatenating all the rows from the CTE table in ascending order, resulting in the final output sequence: "Abi, Rahul, ram, Siva". When executing the query ...
- 868 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Archana_Mathan, did you solve this problem by any chance? I am having a similar problem maintaining the order of the dlt.Thank you,
- 1 kudos
- 839 Views
- 2 replies
- 2 kudos
Resolved! Pandas .to_excel not saving to workspace
Hi!I was always able to export pandas datasets to my workspace on databricks, just like in a local machine using df.to_excel('filename', index=False).but since i got some permissions updates to some new catalogs, when i try to export a dataset using ...
- 839 Views
- 2 replies
- 2 kudos
- 2 kudos
Just to update the situation, and to help anyone with the same problem in the future:What hapened was thas my cluster was updated to use Unity Catalogs, and with that i had to specify the full path to my workspace.df_pandas.to_excel('/Workspace/Users...
- 2 kudos
- 582 Views
- 2 replies
- 1 kudos
Training/Fine-Tunning Foundation model
When I was tring ti run the dbdemos for llm-rag-fine-tuning I'm stuck on one of the first lines of code. When trying to execute the function: databricks.model.foundation_model.create(...) with the Mistral_7B_instruct_v2.0 it is failing with the follo...
- 582 Views
- 2 replies
- 1 kudos
- 1 kudos
Currently using my account and the command dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiUrl().get().apiToken().get() to get my current databricks token to access this function. And my user just doesn't have admin permissions ...
- 1 kudos
- 5435 Views
- 1 replies
- 0 kudos
Identity column has null values
I created a table in databricks using a dbt model pre hook CREATE TABLE IF NOT EXISTS accounts ( account_id BIGINT GENERATED ALWAYS AS IDENTITY, description STRING other columns)I use the same dbt model to merge values into this table in the post...
- 5435 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @rachit-prodigal, Make sure that the account_id column is correctly defined as an identity column in your dbt model. Verify that the GENERATED ALWAYS AS IDENTITY clause is present in the model definition.Confirm that the model is using the correc...
- 0 kudos
- 486 Views
- 1 replies
- 0 kudos
SAP Data in Databricks
Any one using SAP data to brining in to Databricks, do you directly hit SAP ODP or use any third party tool to get the data to datalake . Thanks,
- 486 Views
- 1 replies
- 0 kudos
- 0 kudos
With the recent news form SAP, looks like SAP DataSphere will be the only way forward for, SAP
- 0 kudos
- 530 Views
- 1 replies
- 0 kudos
Query for user persona for DB products
What is the typical user persona for DB products?
- 530 Views
- 1 replies
- 0 kudos
- 0 kudos
Databricks supports many personas, the primary ones are Data Engineer, Data Scientisits, Data Analysts. but also, secondary ones are Analytics Engineers, Machine Learning Engineers, Data Stewareds, Data Administrators and Data Architect.
- 0 kudos
- 541 Views
- 1 replies
- 2 kudos
Resolved! Is there any way to get notified for any new private preview and public preview features
As a databricks customer Is there any way to subscribe to get notified to email for any private preview and public preview features?
- 541 Views
- 1 replies
- 2 kudos
- 2 kudos
Subscribe to the Databricks Newsletter and join the Quarterly Product Roadmap Webinars where they announce all the new private previews.
- 2 kudos
- 406 Views
- 2 replies
- 0 kudos
Error in DLT Streaming table
HelloI am trying to create DLT streaming table in Azure Data bricks but getting following error. I checked online article regarding this error but none of the article shown complete resolution.[STREAM_FAILED] Query [id = 9989a41e-741d-48d7-9a87-fc143...
- 406 Views
- 2 replies
- 0 kudos
- 0 kudos
HI @Kaniz_FatmaCan you pls share the library or package name which contain "com.databricks.cdc.spark.DebeziumJDBCMicroBatchProvider" class. I tried to add databricks-cdc and Debezium library using maven but unable to install them because they are not...
- 0 kudos
- 523 Views
- 1 replies
- 0 kudos
com.databricks.backend.common.rpc.SparkDriverExceptions$SQLExecutionException: java.io.IOException:
I have been getting this error for the past few days while trying to create a table using parquet or csv options. The error is :IOException: s3a://AKIAJBRYNXGHORDHZB4A:a0BzE1bSegfydr3%2FGE3LSPM6uIV5A4hOUfpH8aFF@databricks-corp-training/common/online_...
- 523 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @DMJ, When creating a table, you need to specify the fields (columns) explicitly. In your case, you’re creating a table named outdoorProductsRaw. Make sure to define the columns within the CREATE TABLE statement. For example: CREATE TABLE IF N...
- 0 kudos
- 368 Views
- 1 replies
- 0 kudos
Init_script in workspace
How can use init script stored in workspace which copies jar from storage account and put it inside /databricks/jars location. As said in init_script migration document, where your init scripts are “self-contained,” i.e., DO NOT reference other files...
- 368 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ArvindDige, In Databricks, workspace files are accessible within init scripts, but there are some limitations.When an init script is placed in the workspace, it can only access that specific init script and not other files in the workspace.Ho...
- 0 kudos
- 496 Views
- 2 replies
- 2 kudos
Resolved! Havent receive databricks associate certification
HiI passed my Databricks data engineer associate exam on 28 jun 2024. I haven't recieved my Certification even after 62 hours.Is any body facing the same issue?Tried to contacted databricks bt no response yet.Kindly help
- 496 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Ma2 and @pgoel10, There was a problem issuing certification badges over the weekend. The job is running now and all outstanding badges should be issued by EOD.
- 2 kudos
