- 9255 Views
- 4 replies
- 4 kudos
Resolved! Databricks notebook run
How to run the databricks notebook through ADF ???
- 9255 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Punit Chauhan​ you can use databricks notebook activity in ADF to trigger you databricks notebook via ADF-
- 4 kudos
- 10114 Views
- 3 replies
- 13 kudos
Resolved! Fetching data in excel through delta sharing
Hi all,Is anyway that we can access or push data in delta sharing by using Microsoft excel?
- 10114 Views
- 3 replies
- 13 kudos
- 13 kudos
hey @Ajay Pandey​ yes recently the new excel feature also comes in the market that we can enable the delta sharing from excel also so whatever the changes you will made to delta , it will automaticaly get saved in the excel file also ,refer this lin...
- 13 kudos
- 7029 Views
- 5 replies
- 2 kudos
Resolved! reading multiple csv files using pathos.multiprocessing
I'm using PySpark and Pathos to read numerous CSV files and create many DF, but I keep getting this problem.code for the same:-from pathos.multiprocessing import ProcessingPooldef readCsv(path): return spark.read.csv(path,header=True)csv_file_list =...
- 7029 Views
- 5 replies
- 2 kudos
- 26313 Views
- 3 replies
- 0 kudos
How to get full result using DataFrame.Display method
Hi, Dataframe.Display method in Databricks notebook fetches only 1000 rows by default. Is there a way to change this default to display and download full result (more than 1000 rows) in python? Thanks, Ratnakar.
- 26313 Views
- 3 replies
- 0 kudos
- 0 kudos
display method doesn't have the option to choose the number of rows. Use the show method. It is not neat and you can't do visualizations and downloads.
- 0 kudos
- 30021 Views
- 3 replies
- 1 kudos
- 30021 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Prasanna Lakshmi​ you can use JDBC API to read and write the data from databricks.
- 1 kudos
- 10998 Views
- 4 replies
- 1 kudos
How to merge two separate DELTA LIVE TABLE?
So I have two delta live tables. One that is the master table that contains all the prior data, and another table that contains all the new data for that specific day. I want to be able to merge those two table so that the master table contains would...
- 10998 Views
- 4 replies
- 1 kudos
- 1392 Views
- 0 replies
- 1 kudos
While accessing the data on recipient side using delta_sharing.load_table_changes_as_spark(), it shows data of all versions.
When I tried to access specific version data and set the arguments value to the specific number, I get all version data.data1 = delta_sharing.load_table_changes_as_spark(table_url, starting_version=1, ending_version=1)data2 = delta_sharing.load_table...
- 1392 Views
- 0 replies
- 1 kudos
- 1722 Views
- 0 replies
- 1 kudos
Trouble Displaying Full Size Images from Spark Dataframe
Hi, I have followed this guide (https://learn.microsoft.com/en-us/azure/databricks/_static/notebooks/image-data-source.html) to successfully load some image data into a spark df and display it as a thumbnail. I would like to display a single image fr...
- 1722 Views
- 0 replies
- 1 kudos
- 5866 Views
- 3 replies
- 6 kudos
Resolved! Function When + Dictionary.
Hey everyone, I'm avoiding repeating the When Function for 12x, so I thought of the dictionary. I don't know if it's a limitation of the Spark function or a Logic error. Does the function allow this concatenation?
- 5866 Views
- 3 replies
- 6 kudos
- 6 kudos
Hello everyone, I found this alternative to reduce repeated code.custoDF = (custoDF.withColumn('month', col('Nummes').cast('string')) .replace(months, subset=['month']))
- 6 kudos
- 4101 Views
- 3 replies
- 0 kudos
HTTP 403 on git-credentials API
Hi,I am trying to set git credentials for my service principal. I follow the process described here but I get a 403 error when making the POST request to ${DATABRICKS_HOST}/api/2.0/git-credentials with service principal token.By the way, I also canno...
- 4101 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Sébastien FALQUIER​ it works for me, there are no restrictions. Maybe the PAT token you generated for the service principle got expired. Can you generate a new token and try to run GET/git-credentials API?How are you creating PAT for service prin...
- 0 kudos
- 5110 Views
- 4 replies
- 2 kudos
Cloud provider launch failure
When I want to create a cluster a get this error message:DetailsAWS API error code: InvalidGroup.NotFoundAWS error message: The security group 'sg-0ded75eefd66bf421' does not exist in VPC 'vpc-0ec7da3d5977f6ec9'And when I inspect the security groups ...
- 5110 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi, I am running into a similar issue. but in my case, the security has been deleted by mistake.Is there a way to make Databricks recreate the missing group ?@Kaniz Fatma​ , where can the CreateSecurityGroup command be ran ? Does it change the securi...
- 2 kudos
- 1218 Views
- 0 replies
- 0 kudos
Structured Streaming
I need some solution for below problem.We have set of json files which are keep coming to aws s3, these files contains details for a property . please note 1 property can have 10-12 rows in this json file. Attached is sample json file.We need to read...
- 1218 Views
- 0 replies
- 0 kudos
- 5820 Views
- 4 replies
- 13 kudos
Resolved! To practice Databricks SQL
Is there any sand box kind of thing where we can do some hands-on on Databricks SQL/run the Note books attaching to the Clusters apart from the free trial provided by Databricks.
- 5820 Views
- 4 replies
- 13 kudos
- 13 kudos
Databricks SQL workspace will be available only for Databricks Premium service. If you have Azure Pass subscription, then you can able to get it for practicing it.
- 13 kudos
- 2617 Views
- 4 replies
- 0 kudos
Hi, Has anyone cleared professional DE? please advise on professional data engineer exam. will advance DE learning path be sufficient? Or need to fol...
Hi,Has anyone cleared professional DE? please advise on professional data engineer exam. will advance DE learning path be sufficient? Or need to follow some other resource as well.
- 2617 Views
- 4 replies
- 0 kudos
- 0 kudos
Hello Have a look at this link http://msdatalab.net/how-to-pass-the-professional-databricks-data-engineering/
- 0 kudos
- 2563 Views
- 1 replies
- 0 kudos
Does Databricks provide a mechanism to have rate limiting for receivers?
from pyspark.sql import SparkSession scala_version = '2.12' spark_version = '3.3.0' packages = [ f'org.apache.spark:spark-sql-kafka-0-10_{scala_version}:{spark_version}', 'org.apache.kafka:kafka-clients:3.2.1' ] spark = SparkSession.bui...
- 2563 Views
- 1 replies
- 0 kudos
- 0 kudos
hi @Software Architect​ i dont think so
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 579 | |
| 349 | |
| 287 |
