- 4054 Views
- 4 replies
- 4 kudos
Resolved! Databricks notebook run
How to run the databricks notebook through ADF ???
- 4054 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Punit Chauhan​ you can use databricks notebook activity in ADF to trigger you databricks notebook via ADF-
- 4 kudos
- 7616 Views
- 3 replies
- 13 kudos
Resolved! Fetching data in excel through delta sharing
Hi all,Is anyway that we can access or push data in delta sharing by using Microsoft excel?
- 7616 Views
- 3 replies
- 13 kudos
- 13 kudos
hey @Ajay Pandey​ yes recently the new excel feature also comes in the market that we can enable the delta sharing from excel also so whatever the changes you will made to delta , it will automaticaly get saved in the excel file also ,refer this lin...
- 13 kudos
- 4840 Views
- 5 replies
- 2 kudos
Resolved! reading multiple csv files using pathos.multiprocessing
I'm using PySpark and Pathos to read numerous CSV files and create many DF, but I keep getting this problem.code for the same:-from pathos.multiprocessing import ProcessingPooldef readCsv(path): return spark.read.csv(path,header=True)csv_file_list =...
- 4840 Views
- 5 replies
- 2 kudos
- 21652 Views
- 3 replies
- 0 kudos
How to get full result using DataFrame.Display method
Hi, Dataframe.Display method in Databricks notebook fetches only 1000 rows by default. Is there a way to change this default to display and download full result (more than 1000 rows) in python? Thanks, Ratnakar.
- 21652 Views
- 3 replies
- 0 kudos
- 0 kudos
display method doesn't have the option to choose the number of rows. Use the show method. It is not neat and you can't do visualizations and downloads.
- 0 kudos
- 26227 Views
- 3 replies
- 1 kudos
- 26227 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Prasanna Lakshmi​ you can use JDBC API to read and write the data from databricks.
- 1 kudos
- 8365 Views
- 4 replies
- 1 kudos
How to merge two separate DELTA LIVE TABLE?
So I have two delta live tables. One that is the master table that contains all the prior data, and another table that contains all the new data for that specific day. I want to be able to merge those two table so that the master table contains would...
- 8365 Views
- 4 replies
- 1 kudos
- 955 Views
- 0 replies
- 1 kudos
While accessing the data on recipient side using delta_sharing.load_table_changes_as_spark(), it shows data of all versions.
When I tried to access specific version data and set the arguments value to the specific number, I get all version data.data1 = delta_sharing.load_table_changes_as_spark(table_url, starting_version=1, ending_version=1)data2 = delta_sharing.load_table...
- 955 Views
- 0 replies
- 1 kudos
- 1230 Views
- 0 replies
- 1 kudos
Trouble Displaying Full Size Images from Spark Dataframe
Hi, I have followed this guide (https://learn.microsoft.com/en-us/azure/databricks/_static/notebooks/image-data-source.html) to successfully load some image data into a spark df and display it as a thumbnail. I would like to display a single image fr...
- 1230 Views
- 0 replies
- 1 kudos
- 3985 Views
- 3 replies
- 6 kudos
Resolved! Function When + Dictionary.
Hey everyone, I'm avoiding repeating the When Function for 12x, so I thought of the dictionary. I don't know if it's a limitation of the Spark function or a Logic error. Does the function allow this concatenation?
- 3985 Views
- 3 replies
- 6 kudos
- 6 kudos
Hello everyone, I found this alternative to reduce repeated code.custoDF = (custoDF.withColumn('month', col('Nummes').cast('string')) .replace(months, subset=['month']))
- 6 kudos
- 2775 Views
- 3 replies
- 0 kudos
HTTP 403 on git-credentials API
Hi,I am trying to set git credentials for my service principal. I follow the process described here but I get a 403 error when making the POST request to ${DATABRICKS_HOST}/api/2.0/git-credentials with service principal token.By the way, I also canno...
- 2775 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Sébastien FALQUIER​ it works for me, there are no restrictions. Maybe the PAT token you generated for the service principle got expired. Can you generate a new token and try to run GET/git-credentials API?How are you creating PAT for service prin...
- 0 kudos
- 3485 Views
- 4 replies
- 2 kudos
Cloud provider launch failure
When I want to create a cluster a get this error message:DetailsAWS API error code: InvalidGroup.NotFoundAWS error message: The security group 'sg-0ded75eefd66bf421' does not exist in VPC 'vpc-0ec7da3d5977f6ec9'And when I inspect the security groups ...
- 3485 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi, I am running into a similar issue. but in my case, the security has been deleted by mistake.Is there a way to make Databricks recreate the missing group ?@Kaniz Fatma​ , where can the CreateSecurityGroup command be ran ? Does it change the securi...
- 2 kudos
- 830 Views
- 0 replies
- 0 kudos
Structured Streaming
I need some solution for below problem.We have set of json files which are keep coming to aws s3, these files contains details for a property . please note 1 property can have 10-12 rows in this json file. Attached is sample json file.We need to read...
- 830 Views
- 0 replies
- 0 kudos
- 3928 Views
- 4 replies
- 13 kudos
Resolved! To practice Databricks SQL
Is there any sand box kind of thing where we can do some hands-on on Databricks SQL/run the Note books attaching to the Clusters apart from the free trial provided by Databricks.
- 3928 Views
- 4 replies
- 13 kudos
- 13 kudos
Databricks SQL workspace will be available only for Databricks Premium service. If you have Azure Pass subscription, then you can able to get it for practicing it.
- 13 kudos
- 13043 Views
- 3 replies
- 3 kudos
Resolved! UUID/GUID Datatype in Databricks SQL
Hi all,I am trying to create a table with a GUID column.I have tried using GUID, UUID; but both of them are not working.Can someone help me with the syntax for adding a GUID column?Thanks!
- 13043 Views
- 3 replies
- 3 kudos
- 3 kudos
Hey @Avinash Narasimhan​ , What is the exact problem you are getting can you please share it is working fine for meThanksAviral Bhardwaj
- 3 kudos
- 1839 Views
- 4 replies
- 0 kudos
Hi, Has anyone cleared professional DE? please advise on professional data engineer exam. will advance DE learning path be sufficient? Or need to fol...
Hi,Has anyone cleared professional DE? please advise on professional data engineer exam. will advance DE learning path be sufficient? Or need to follow some other resource as well.
- 1839 Views
- 4 replies
- 0 kudos
- 0 kudos
Hello Have a look at this link http://msdatalab.net/how-to-pass-the-professional-databricks-data-engineering/
- 0 kudos
| User | Count |
|---|---|
| 1612 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
