- 1875 Views
- 1 replies
- 0 kudos
Exceptions are Not Getting Handled In Autoloader Write Stream
I have below logic implemented using Databricks Autoloader. ## Autoloader Write stream: Its calling forEachBatch function to write into respective datatype catalog table# and using checkpoint to keeps track of processing files.try: ##Observe raw ...
- 1875 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Sambit_S ,In your scenario, there is a merge failure. Your query won't be able to progress as the problematic batch can't be committed to sink. Even if you handle the exception in a try catch block, it's impossible for the autoloader to update...
- 0 kudos
- 3242 Views
- 1 replies
- 0 kudos
Increase stack size Databricks
Hi everyoneI'm currently running a shell script in a notebook, and I'm encountering a segmentation fault. This is due to the stack size limitation. I'd like to increase the stack size using ulimit -s unlimited, but I'm facing issues with setting this...
- 3242 Views
- 1 replies
- 0 kudos
- 18275 Views
- 5 replies
- 0 kudos
How to connect Databricks Database with Springboot application using JPA
facing issue with integrating our Spring boot JPA supported application with Databricks.Below are the steps and setting we did for the integration.When we are starting the spring boot application we are getting a warning as :HikariPool-1 - Driver doe...
- 18275 Views
- 5 replies
- 0 kudos
- 0 kudos
Was there any resolution to this? Is Spring datasource supported now?
- 0 kudos
- 7319 Views
- 2 replies
- 1 kudos
Resolved! How to get a list of workspace users who have the "unrestricted cluster create" entitlement ?
Hello - I'm hoping somebody can help me with this ... I have a lot of users configured with access to a workspace (100's) and I want to write a report to see if any of the users have "unrestricted cluster create" entitlement in the workspace. This i...
- 7319 Views
- 2 replies
- 1 kudos
- 1 kudos
Many thanks for you help @Yeshwanth it put me on the right track. The API does have a filter option and that looks like it complies with rfc7644 but my attempts to use it were rather hit and miss - I suspect as the API is preview it is not fully imp...
- 1 kudos
- 12629 Views
- 11 replies
- 2 kudos
Sql Serverless Option is missing when using Azure Databricks Workspace with No Public IP and VNET Injection
HelloAfter creating an Databricks Workspace in Azure with No Public IP and VNET Injection, I'm unable to use DBSQL Serverless because the option to enable it in SQL warehouse Settings is missing. ​Is it by design? Is it a limitation when using Privat...
- 12629 Views
- 11 replies
- 2 kudos
- 2 kudos
Fixed, go to Profile -> Compute-> SQL Server Serverless -> On -> Save. For some reason this has been disabled for us.
- 2 kudos
- 7812 Views
- 1 replies
- 0 kudos
Resolved! R install - cannot open URL
Neither standard nor non standard repo seem available. Any idea how to debug/fix this? %r install.packages("gghighlight", lib="/databricks/spark/R/lib", repos = "http://cran.us.r-project.org") Warning: unable to access index for repository http://cra...
- 7812 Views
- 1 replies
- 0 kudos
- 0 kudos
%sh nc -zv cran.us.r-project.org 80 It was a network issue. These lines above proved it and the network administrators had to open the IPs.
- 0 kudos
- 6758 Views
- 4 replies
- 0 kudos
Delta Live Tables with Common Data Model as source
Hi,I'm investigating whether it's possible to use Common Data Model CDM (in particular the Dynamics 365 exported csv and cdm files) as a Delta Live Tables data source? Can someone point me in the right direction?Thanks!
- 6758 Views
- 4 replies
- 0 kudos
- 0 kudos
Using Delta Live Tables with Common Data Model (CDM) as a Source in DatabricksI'm investigating the use of Delta Live Tables (DLT) to process Common Data Model (CDM) files exported from Dynamics 365, and I found a solution that works well. Here’s a q...
- 0 kudos
- 2017 Views
- 3 replies
- 1 kudos
get job detail API
Hello, is there an API interface for passing in batches of run_id to obtain job running details?
- 2017 Views
- 3 replies
- 1 kudos
- 1 kudos
Maybe this could help. Its not batch, but you can get the run_id details https://docs.databricks.com/en/workflows/jobs/jobs-2.0-api.html#runs-get-output
- 1 kudos
- 2207 Views
- 1 replies
- 0 kudos
Autoloader with SQLite db files
Hi Everyone, Is it possible to ingest SQLite db files with Databricks Autoloader? Is it currently supported? If so, could you please share an example?
- 2207 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @eva_mcmf , I hope this message finds you well. As per the documentation, Auto Loader incrementally and efficiently processes new data files as they arrive in cloud storage. Auto Loader can load data files from AWS S3, Azure Data Lake Storage G...
- 0 kudos
- 4803 Views
- 1 replies
- 0 kudos
The default location of temporary file in Azure Synapse Connector(com.databricks.spark.sqldw)
Hi everone, I'm trying to query data in Azure Synapse Dedicated SQL Pool according to the documentaion using:.format("com.databricks.spark.sqldw") Query data in Azure Synapse AnalyticsIt says that a abfss temporary location is needed.But I found that...
- 4803 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Chengcheng , I hope this message finds you well. As per the documentation the "tempDir" parameter is a required one and there is no default value for it.Databricks Synapse connector options reference: https://docs.databricks.com/en/connect/ext...
- 0 kudos
- 4235 Views
- 4 replies
- 1 kudos
Resolved! My Libraries are not being installed in dbx-pipelines
Hello,I have some libraries on Azure Artifacts, but when I'm using notebooks, they are unreachable even though I'm explicitly adding the pip extra-url option (I have validated the tokens). So, I had to install them manually by downloading the wheel f...
- 4235 Views
- 4 replies
- 1 kudos
- 1 kudos
@shan_chandrawe solved it, it was an issue with the DevOps key-vault token associated of the artifacts token.
- 1 kudos
- 1464 Views
- 1 replies
- 0 kudos
Resolved! Delta Lake Table Daily Read and Write job optimization
I have created 7 job for each business system to extract product data from each postgress source then write all job data into one data lake delta table [raw_product].each business system product table has around 20 GB of data.do the same thing for 15...
- 1464 Views
- 1 replies
- 0 kudos
- 0 kudos
@AH - we can try out the config if read or fetch from postgres is slow , we can increase the fetchsize , numPartitions (to increase parallelism). kindly try to do a df.count() to check on slowness. https://spark.apache.org/docs/latest/sql-data-sou...
- 0 kudos
- 15644 Views
- 4 replies
- 0 kudos
Resolved! Using parameters in a SQL Notebook and COPY INTO statement
Hi, My scenario is I have an export of a table being dropped in ADLS every day. I would like to load this data into a UC table and then repeat the process every day, replacing the data. This seems to rule out DLT as it is meant for incremental proc...
- 15644 Views
- 4 replies
- 0 kudos
- 0 kudos
The solution that worked what adding this python cell to the notebook: %pythonfrom pyspark.dbutils import DBUtilsdbutils = DBUtils(spark)dbutils.widgets.text("catalog", "my_business_app")dbutils.widgets.text("schema", "dev") Then in the SQL Cell: CRE...
- 0 kudos
- 2786 Views
- 1 replies
- 0 kudos
Job running time too long
Hi all,I'm doing my first data jobs.I create one job that consists of 4 other jobs.Yesterday I ran the 4 jobs separately and it worked fine (about half hour)-today I ran the big job, and the 4 jobs is running for 2 hours (and still running), Why is t...
- 2786 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @galzamo how are you? You can check on the SparkUI for long running stages that might give you a clue where it's spending the most time on each task. Somethings can be the reason: 1. Increase of data and partitions on your source data 2. Cluste...
- 0 kudos
- 2522 Views
- 1 replies
- 0 kudos
Expected size of managed Storage Accounts
Dear all,we are monitoring the size of managed storage accounts associated with our deployed Azure databricks instances.We have 5 databricks instances for specific components of our platform replicated in 4 environments (DEV, TEST, PREPROD, PROD).Dur...
- 2522 Views
- 1 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 581 | |
| 349 | |
| 287 |
