- 691 Views
- 1 replies
- 0 kudos
Data Engineering Hub
I have been using databricks from last 2 years and have got involved with arranging when I was in my masters. Databricks did not made data engineering easy for me but helped me become a data engineer. I got into this 2 all day training from Advanced ...
- 691 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @AbbasNvs, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Community T...
- 0 kudos
- 1397 Views
- 5 replies
- 0 kudos
DataBricks Certification Exam Got Suspended. Require support for the same.
Hello Team, I encountered Pathetic experience while attempting my 1st DataBricks certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam without giving any reaso...
- 1397 Views
- 5 replies
- 0 kudos
- 0 kudos
@Kaniz @Cert-Team @Sujitha I have sent multiple emails to the Support team to reschedule my exam with Date, but I have not received any confirmation from them.Please look into this issue and reschedule the exam as soon as possible. This certification...
- 0 kudos
- 1037 Views
- 1 replies
- 1 kudos
capture return value from databricks job to local machine by CLI
Hi,I want to run a python code on databricks notebook and return the value to my local machine. Here is the summary:I upload files to volumes on databricks. I generate a md5 for local file. Once the upload is finished, I create a python script with t...
- 1037 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @pshuk, You could check the below CLI commands: get-run-output Get the output for a single run. This is the REST API reference, which relates to the CLI command: https://docs.databricks.com/api/workspace/jobs/getrunoutput export-run There's al...
- 1 kudos
- 1778 Views
- 2 replies
- 0 kudos
Resolved! Error Code: METASTORE_DOES_NOT_EXIST when using Databricks API
Hello, I'm attempting to use the databricks API to list the catalogs in the metastore. When I send the GET request to `/api/2.1/unity-catalog/catalogs` , I get this error I have checked multiple times and yes, we do have a metastore associated with t...
- 1778 Views
- 2 replies
- 0 kudos
- 0 kudos
Turns out I was using the wrong databricks host url when querying from postman. I was using my Azure instance instead of my AWS instance.
- 0 kudos
- 1261 Views
- 1 replies
- 0 kudos
Parallel kafka consumer in spark structured streaming
Hi,I have a spark streaming job which reads from kafka and process data and write to delta lake.Number of kafka partition: 100number of executor: 2 (4 core each)So we have 8 cores total which are reading from 100 partitions of a topic. I wanted to un...
- 1261 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @subham0611, In Spark Streaming, the number of threads is not explicitly controlled by the user. Instead, the parallelism is determined by the number of partitions in the Kafka topic. Each partition is consumed by a single Spark task. When you ...
- 0 kudos
- 914 Views
- 1 replies
- 1 kudos
how to develop Notebooks on vscode for git repos?
I am able to use vscode extension + databricks connect to develop Notebooks on my local computer and run them on my databricks cluster. However I can not figure out how to develop the Notebooks that have the file `.py` extension but identified by Dat...
- 914 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @MohsenJ, You can indeed develop and run Databricks notebooks with .py extension on your local computer using the Databricks extension for Visual Studio Code and Databricks Connect1. Here are the steps you need to follow: Enable Databricks Con...
- 1 kudos
- 573 Views
- 1 replies
- 0 kudos
Assessment(Assessment job need to be deployed using Terraform)
Assessment(Assessment job need to be deployed using Terraform)1.Install latest version of UCX 2.UCX will add the assessment job and queries to the workspace3.Run the assessment using ClusterHow to write code for this by using Terraform. Can anyone he...
- 573 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @unity_Catalog, To deploy an assessment job using Terraform, you’ll need to write a Terraform configuration that includes the following steps: Install the latest version of UCX: This will likely involve using a provider to manage software inst...
- 0 kudos
- 10700 Views
- 4 replies
- 4 kudos
Resolved! Use SQL Server Management Studio to Connect to DataBricks?
The Notebook UI doesn't always provide the best experience for running exploratory SQL queries. Is there a way for me to use SQL Server Management Studio (SSMS) to connect to DataBricks? See Also:https://learn.microsoft.com/en-us/answers/questions/74...
- 10700 Views
- 4 replies
- 4 kudos
- 4 kudos
What you can do is define a SQL endpoint as a linked server. Like that you can use SSMS and T-SQL.However, it has some drawbacks (no/bad query pushdown, no caching).Here is an excellent blog of Kyle Hale of databricks:Tutorial: Create a Databricks S...
- 4 kudos
- 1286 Views
- 5 replies
- 3 kudos
I am facing an issue while generating the DBU consumption report and need help.
I am trying to access the following system tables to generate a DBU consumption report, but I am not seeing this table in the system schema. Could you please help me how to access it?system.billing.inventory, system.billing.workspaces, system.billing...
- 1286 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi @Cloud_Architect, In Azure Databricks, system tables are an analytical store of your account’s operational data found in the system catalog1. These tables can be used for historical observability across your account1. To access system tables, y...
- 3 kudos
- 988 Views
- 1 replies
- 2 kudos
ingest csv file on-prem to delta table on databricks
Hi,So I want to create a delta live table using a csv file that I create locally (on-prem). A little background: So I have a working ELT pipeline that finds newly generated files (since the last upload), and upload them to databricks volume and at th...
- 988 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @pshuk , Based on your description, you have an external pipeline that writes CSV files to a specific storage location and you wish to set up a DLT based on the output of this pipeline. DLT offers has access to a feature called Autoloader, whic...
- 2 kudos
- 1501 Views
- 3 replies
- 0 kudos
Resolved! Delta Sharing - Info about Share Recipient
What information do you know about a share recipient when they access a table shared to them via Delta Sharing?Wondering if we might be able to utilize something along the lines of is_member, is_account_group_member, session_user, etc for ROW and COL...
- 1501 Views
- 3 replies
- 0 kudos
- 0 kudos
Now that I'm looking closer at the share credentials and the recipient entity you would really need a way to know the bearer token and relate that back to various recipient properties - databricks.name and any custom recipient property tags you may h...
- 0 kudos
- 1372 Views
- 2 replies
- 2 kudos
Updating Databricks SQL Warehouse using Terraform
We can Update SQL Warehouse manually in Databricks.Click SQL Warehouses in the sidebarIn Advanced optionsWe can find Unity Catalog toggle button there! While Updating Existing SQL Warehouse in Azure to enable unity catalog using terraform, I couldn'...
- 1372 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Sudheer2, The Unity Catalog is a feature in Databricks SQL Warehouse that allows you to query data across multiple databases and tables seamlessly. It provides a unified view of your data.When you enable the Unity Catalog, you can access tables f...
- 2 kudos
- 3364 Views
- 7 replies
- 1 kudos
log signature and input data for Spark LinearRegression
I am looking for a way to log my `pyspark.ml.regression.LinearRegression` model with input and signature ata. The usual example that I found around are using sklearn and they can simply do # Log the model with signature and input example signature =...
- 3364 Views
- 7 replies
- 1 kudos
- 1 kudos
me neither. But the mlflow documentation suggests the new version of mlflow should be able to handle Array and Objects (dict). maybe that could help? I haven't tired it myself. Support for Array and Object types was introduced in MLflow version 2.10...
- 1 kudos
- 1395 Views
- 1 replies
- 0 kudos
Data analytics
Really excited that Data bricks is championing the cause of unifying the whole data analytics value chain. it’s a great service to the data community.
- 1395 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Rpswadhwa, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Communit...
- 0 kudos
- 709 Views
- 1 replies
- 0 kudos
2023 Data AI summit
Have learned a lot through this experience, especially around productionalizing Gen AI models. Great time and great people!
- 709 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @BDietrich, We're thrilled to hear that you had a great experience at DAIS 2023! Your feedback is valuable to us, and we appreciate you taking the time to share it on the community platform. We wanted to let you know that the Databricks Communit...
- 0 kudos
