Is it possible to execute sql server agent job from Databricks notebook?
i dont think this type of feature is available there
I am using DBR 10.4 LTS instance can anyone help me formatting the code.I have tried with format python error pop up with upgrade to DBR 11.2 any other alternative to this?
please give us a code by that we can help you
I am running a java/jar Structured Streaming job on a single node cluster (Databricks runtime 8.3). The job contains a single query which reads records from multiple Azure Event Hubs using Spark Kafka functionality and outputs results to a mssql dat...
its seems that when your nodes are increasing it is seeking for init script and it is failing so you can use reserve instances for this activity instead of spot instances it will increase your overall costor alternatively, you can use depended librar...
I am configuring an Databricks jobs using multiple notebooks having dependency with each other. All the notebooks are parameterized and using similiar parameters. How can i configure the parameter on global level so that all the notebooks can consume...
actually, it is very hard but if you want to use an alternative option you have to change your code and use a widget feature of data bricks.May be this is not a right option but you can still explore this doc for testing purpose https://docs.databric...
Hello,I need to schedule some of my jobs within Databricks Workflow every other week (or every 4 weeks). I've scoured a few forums for find what this notation would be, but I've been unfruitful in my searches.Is this scheduling possible in crontab? I...
For every seven days starting from Monday, you need to use 2/7. From my experience, that generator works best with databricks https://www.freeformatter.com/cron-expression-generator-quartz.html
Can anyone let me know, Is there anyway In which we can access different workspace delta tables in a workspace where we run the pipelines using python?​
@Hemanth A​ go to the workspace you want data from, in warehouse tab you will find connectivity in that copy host name, http path and generate token for it, by this credentials you can access the data of this workspace in any other workspace.
I need to count the number of campaigns per day based on the start and end dates of the campaignsInput Table: Out needed (result):How do I need to write the SQL command in databricks to get the above result? thanks all
Just create an array with sequence, explode it, and then group and count:WITH cte AS (SELECT `campaign name`, explode(sequence(`Start date`, `End date`, interval 1 day)) as `Date` FROM `campaigns`) SELECT Count(`campaign name`) as `count uni...
SELECT '(CC) ABC' REGEXP '\\b\\(CC\\)\\b' AS TEST1, 'A(CC) ABC' REGEXP '\\b\\(CC\\)\\b' AS TEST2, 'A (CC)A ABC' REGEXP '\\b\\(CC\\)\\b' AS TEST3, 'A (CC) A ABC' REGEXP '\\b\\(CC\\)\\b' AS TEST4, 'A ABC (CC)' REGEXP '\\b\\(CC\\)\\b' AS TES...
I'm able to make it to the Permission page of the schema and table I'm trying to do access control on within the Data Explorer page.At first you can only grant permissions but not revoke anything. Only after you have made new grants can you revoke w...
if I manually delete some parque files in location which the real data is stored in, so spark catalog still has the old version. How can I sync them?Thanks!
You just need to create a new table and specify the location of the data for your case it's going to be an ADLS, S3...Example​Create table customer using delta location 'mnt/data./'
I have a custom application/executable that I upload to DBFS and transfer to my cluster's local storage for execution. I want to call multiple instances of this application in parallel, which I've only been able to successfully do with Python's subpr...
Autoscaling works for spark jobs only. It works by monitoring the job queue, which python code won't go into. If it's just python code, try single node.https://docs.databricks.com/clusters/configure.html#cluster-size-and-autoscaling
You can find a rich ecosystem of tools that allow you to work with all your data in-place and deliver real-time business insights faster.This post will help you connect your existing tools like dbt, Fivetran, PowerBI, Tableau or SAP to ingest, transf...
Hello Taha, here is a fairly recent video provided by Databricks on conncecting Power BI : Demo Video: Connect to Power BI Desktop from Databricks - YouTube
Hi All,Hope everyone is doing well.We are currently validating Databricks on GCP and Azure.We have a python notebook that does some ETL (Copy, extract zip files and process files within the zip files)Our Cluster Config on AzureDBX Runtime - 10.4 - Dr...
hi @Tunde Abib​ , I have gone through the links while updating, but did not see any major documented slow downs mentioned in them.
KB Feedback Discussion In addition to the Databricks Community, we have a Support team that maintains a Knowledge Base (KB). The KB contains answers to common questions about Databricks, as well as information on optimisation and troubleshooting.Thes...
Thanks for sharing @Sujitha Ramamoorthy​
I had been trying to upsert rows into a table in Azure Blob Storage (ADLS Gen 2) based on two partitions (sample code below). insert overwrite table new_clicks_table partition(client_id, mm_date) select click_id ,user_id ,click_timestamp_gmt ,ca...
Below code might help youPython- (df.write .mode("overwrite") .option("partitionOverwriteMode", "dynamic") .saveAsTable("default.people10m") ) SQL- SET spark.sql.sources.partitionOverwriteMode=dynamic; INSERT OVERWRITE TABLE default.people10m...
| User | Count |
|---|---|
| 1601 | |
| 736 | |
| 343 | |
| 284 | |
| 247 |
