- 4921 Views
- 3 replies
- 4 kudos
Resolved! Load data into Azure SQL Database from Azure Databricks ( restricted table not a whole workspace tables)
Hi ,I want to share limited tables in my databricks workspace and users will connect to my databricks through Azure Data factory and will load data into Azure SQL. Is this possible using Delta Sharing? Or any other method or tool?
- 4921 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Dayananthan Marimuthu​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your...
- 4 kudos
- 4493 Views
- 3 replies
- 3 kudos
Unable to find input by typing input in the Multiselect Widget
In the AWS databricks widgets.multiselect, I'm unable to find input by typing input in the mulitselect bar. It was working before. Although I can find the inputs by scrolling down the list, it's annoying if the list is long.​​Here's my script:measlis...
- 4493 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Philip Teu​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 3 kudos
- 8013 Views
- 3 replies
- 4 kudos
Resolved! Confusion in string comparison
Hello expertI am new to spark. I am using same price of code but getting different resultsWhen i am using below piece of code, i am getting errorpy4j.Py4JException: Method or([class java.lang.String]) does not existdf.filter(F.col("state").isNull() ...
- 8013 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Saswata Dutta​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedbac...
- 4 kudos
- 6841 Views
- 2 replies
- 3 kudos
Databricks driver logs
I am running jobs on databricks clusters. When the cluster is running I am able to find the executor logs by going to Spark Cluster UI Master dropdown, selecting a worker and going through the stderr logs. However, once the job is finished and cluste...
- 6841 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Atul Arora​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback w...
- 3 kudos
- 9377 Views
- 2 replies
- 2 kudos
How do I configure managed identity to databricks cluster and access azure storage using spark config
Partner want to use adf managed identity to connect to my databricks cluster and connect to my azure storage and copy the data from my azure storage to their azure storage storage
- 9377 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @SAI PUSALA​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback w...
- 2 kudos
- 3595 Views
- 2 replies
- 2 kudos
Failure to initialize configuration' on SQL Warehouse Tables
Yesterday I had a basic DLT pipeline up and running, and was able to query the hive_metastore tables successfully. The pipeline uses autloader to ingest a few csv files from cloud storage to streaming live bronze and silver tables. Today after star...
- 3595 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Jennette Shepard​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feed...
- 2 kudos
- 2831 Views
- 2 replies
- 4 kudos
Resolved! DLT code examples and notebooks?
we like the examples that you show in webinars especially with DLT and Huggingface or DLT with ingestion from Kafka, are they publicly available?
- 2831 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi @Frank Munz​ Thank you for your question! To assist you better, please take a moment to review the answer and let me know if it best fits your needs.Please help us select the best solution by clicking on "Select As Best" if it does.Your feedback w...
- 4 kudos
- 4806 Views
- 4 replies
- 3 kudos
- 4806 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Vijay Gadhave​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 3 kudos
- 2923 Views
- 2 replies
- 2 kudos
To collect the elements of a SparkDataFrame and coerces them into an R dataframe.
Hello Everyone,I am facing the challenge while collecting a spark dataframe into an R dataframe, this I need to do as I am using TraMineR algorithm whih is implemented in R only and the data pre-processing I have done in pysparkI am trying this:event...
- 2923 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Niraj Tanwar​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 2 kudos
- 3964 Views
- 4 replies
- 3 kudos
Automating the initial configuration of dbx
Hi Team,Good morning.As of now, for the deployment of our code to Databricks, dbx is configured providing the parameters such as cloud provider, git provider, etc., Say, I have a code repository in any one of the git providers. Can this process of co...
- 3964 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi @Arunsundar Muthumanickam​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear fr...
- 3 kudos
- 8602 Views
- 4 replies
- 1 kudos
Resolved! How to set properties for a delta table when I want to write a DataFrame?
Hi,I have a PySpark DataFrame with 11 million records. I created the DataFrame on a cluster. It is not saved on DBFS or storage account. import pyspark.sql.functions as F from pyspark.sql.functions import col, when, floor, expr, hour, minute, to_time...
- 8602 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Mohammad Saber​ , Are you getting the error while writing the file to the table? Or before that?
- 1 kudos
- 9927 Views
- 2 replies
- 2 kudos
Resolved! FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.
I'm trying to get rid of the warning below:/databricks/spark/python/pyspark/sql/context.py:117: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead.In my setup, I have a front-end notebook that gets parameters from the ...
- 9927 Views
- 2 replies
- 2 kudos
- 2 kudos
That fixes it. Thanks. I need to do spark = SparkSession.builder.getOrCreate() df = spark.table("prod.some_schema.some_table")instead of sc = SparkSession.builder.getOrCreate() sqlc = SQLContext(sc) df = sqlc.table(f"prod.some_schema.some...
- 2 kudos
- 9253 Views
- 1 replies
- 3 kudos
Resolved! Set Workflow Job Concurrency Limit
Hi Everyone,I need a job to be triggered every 5 minutes. However, if that job is already running, it must not be triggered again until that run is finished. Hence, I need to set the maximum run concurrency for that job to only one instance at a time...
- 9253 Views
- 1 replies
- 3 kudos
- 3 kudos
@Michael Okulik​ :To ensure that a Databricks job is not triggered again until a running instance of the job is completed, you can set the maximum concurrency for the job to 1. Here's how you can configure this in Databricks:Go to the Databricks work...
- 3 kudos
- 3566 Views
- 3 replies
- 0 kudos
Databricks Spark certification voucher code expired
Hi Team,I am getting error that voucher code expired when trying to register for "Databricks Certified Associate Developer for Apache Spark 3.0 - Python" certification.Can you please help here
- 3566 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Sandeep Venishetti​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell ...
- 0 kudos
- 17154 Views
- 3 replies
- 0 kudos
- 17154 Views
- 3 replies
- 0 kudos
- 0 kudos
Databricks provides 4 main deployment models they are:Public Cloud Deployment Model: Databricks can be deployed on public cloud platforms such as AWS, Azure, and Google Cloud Platform. This is the most common deployment model for Databricks and provi...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 554 | |
| 349 | |
| 287 |
