- 7029 Views
- 5 replies
- 1 kudos
Model Training Data Adapter Error.
We are converting Pyspark dataframe to Tensorflow using PetaStorm and have encountered a “data adapter” error. What do you recommend for diagnosing and fixing this error?https://docs.microsoft.com/en-us/azure/databricks/applications/machine-learning/...
- 7029 Views
- 5 replies
- 1 kudos
- 1 kudos
Hey @Nathan Law Thank you so much for getting back to us. We will await your response.We really appreciate your time.
- 1 kudos
- 3956 Views
- 1 replies
- 2 kudos
Illegal character in partition path when attempting REORG ... (PURGE)
I have a large delta table partitioned by an identifier column that I now have discovered has blank spaces in some of the identifiers, e.g. one partition can be defined by "Identifier=first identifier". Most partitions does not have these blank space...
- 3956 Views
- 1 replies
- 2 kudos
- 2 kudos
FYI similar issue with partitions with "%" in the identifier. Used the filter clause of the REORG to exclude partitions with " " or "%" to be able to move forward with my work but will continue looking for a solution. I've never seen any pointers not...
- 2 kudos
- 30976 Views
- 12 replies
- 13 kudos
Resolved! Failed to convert Spark.sql to Pandas Dataframe using .toPandas()
I wrote the following code:data = spark.sql (" SELECT A_adjClose, AA_adjClose, AAL_adjClose, AAP_adjClose, AAPL_adjClose FROM deltabase.a_30min_delta, deltabase.aa_30min_delta, deltabase.aal_30min_delta, deltabase.aap_30min_delta ,deltabase.aapl_30m...
- 30976 Views
- 12 replies
- 13 kudos
- 13 kudos
I just discovered a solution.Today, I opened Azure Databricks. When I imported python libraries. Databricks told me that toPandas() was deprecated and it suggested me to use toPandas.The following solution works: Use toPandas instead of toPandas() da...
- 13 kudos
- 5512 Views
- 3 replies
- 3 kudos
Resolved! Git integration for enterprises with a private git server behind VPN
The documentation states that DataBricks does not support private Git servers behind a VPN. The forum does however state in two places (place1, place2) that enterprise customers can reach out to their 'account team' and request to be added to somethi...
- 5512 Views
- 3 replies
- 3 kudos
- 3 kudos
Hey all, What do you say about the below post. I am little bit confused about it. If someone will help me, it will be appreciated. https://community.databricks.com/s/question/0D53f00001GHVYnCAP/will-databricks-support-selfservice-web-application-fire...
- 3 kudos
- 17042 Views
- 2 replies
- 5 kudos
How to understand what dropDuplicates is doing?
Smashed our heads against this one for a while and though I think it’s more of a spark question than a Databricks one, wanting to get your thoughts on it. Essentially the gist is this:We select into a DF from a delta tableWe display the DF and see 2 ...
- 17042 Views
- 2 replies
- 5 kudos
- 5 kudos
Great answer @Aman Sehgal. I also received another answer from @Ryan Chynoweth I will paste here:1) Have you seen anything like this before and if so, can you provide any insight on it?Yes this does happen due to the lazy execution of spark and due...
- 5 kudos
- 779 Views
- 0 replies
- 3 kudos
Hello again Databricks Community! On July 28th we are hosting another Community Social Event! We want to make sure that we all have the chance to con...
Hello again Databricks Community! On July 28th we are hosting another Community Social Event! We want to make sure that we all have the chance to connect as a community often. Come network, talk data, and just get social! Join us for our July Communi...
- 779 Views
- 0 replies
- 3 kudos
- 2472 Views
- 1 replies
- 0 kudos
Failed to start cluster.
Hello,i'm getting this message today everytime i try to run a notebook since i logged in:When i click 'ok' nothing happens.I tried refreshing and switching browsers already. Tried run this cell, run below cells, run all cells, etc. I also tried to cr...
- 2472 Views
- 1 replies
- 0 kudos
- 690 Views
- 0 replies
- 0 kudos
Structured Streaming migration from readStream to Auto Loader (AWS)
Hello!I have a (non-stateful) daily structured streaming ingestion job from json S3 file storage into a Delta table using Spark readStream. I would like to switch to Auto Loader to reduce file discovery time. My questions are:Do I need to create a ne...
- 690 Views
- 0 replies
- 0 kudos
- 3001 Views
- 2 replies
- 0 kudos
Connecting Power BI to DBSQL
Trying to connect Power BI with veresion 2.106.582.0 32 bit But get the error
- 3001 Views
- 2 replies
- 0 kudos
- 0 kudos
@Wilbur Tong Along with the steps mentioned by @Akash Bhat please try to follow the steps mentioned in below document:https://docs.microsoft.com/en-us/power-bi/connect-data/desktop-connector-extensibility#custom-connectorsfor downloading the pqx fi...
- 0 kudos
- 2141 Views
- 0 replies
- 0 kudos
TheINFO.pk is the premier and most trustworthy source of information in Pakistan. We provides latest news, updates, jobs, Results, Education, telecom ...
TheINFO.pk is the premier and most trustworthy source of information in Pakistan. We provides latest news, updates, jobs, Results, Education, telecom and many more stuff those you love.
- 2141 Views
- 0 replies
- 0 kudos
- 20306 Views
- 6 replies
- 9 kudos
Error with databricks workspace
Hi,I have the following error :Error: b'{"error_code":"TEMPORARILY_UNAVAILABLE","message":"The service at /api/2.0/workspace/get-status is temporarily unavailable. Please try again later."}'when I do :databricks workspace export_dir path .ordatabrick...
- 20306 Views
- 6 replies
- 9 kudos
- 9 kudos
Please try to reconfigure cli. Please double check databricks hostdatabricks configure --tokenRegarding second command which you shared (%sh ls /Workspace) it will not work on free community edition. There you can use only native function like - dbu...
- 9 kudos
- 5631 Views
- 6 replies
- 7 kudos
Data & AI Summit 2022 - Training Videos of paid Instructor led sessions not yet uploaded. @kaniz fatma
@Kaniz Fatma I attended the Advanced Machine Learning with Databricks training last week virtually I am still unable to get the day 2 session videos of any of the Instructor led Paid Trainings. They are supposed to be available for replay with in 24...
- 5631 Views
- 6 replies
- 7 kudos
- 7 kudos
Hi @Kaniz Fatma , they uploaded the full video for Advanced Machine Learning with Databricks course day 2, Thank you for the follow up. but still we have the same issue with Apache Spark Programming with Databricks - Bundle: Day 2 Training . can you...
- 7 kudos
- 4305 Views
- 2 replies
- 1 kudos
Resolved! Finding multiple substrings from a DataFrame column dynamically?
Hello friends,I have a DataFrame with specific values. I am trying to find specific values out of it. *I/P -|ID | text ||:--|:------||1 | select distinct Col1 as OrderID from Table1 WHERE ( (Col3 Like '%ABC%') OR (Col3 Like '%DEF%') OR (Col3 Like '...
- 4305 Views
- 2 replies
- 1 kudos
- 1 kudos
What is the logic for substring function?Can't you use str1[idxi+14:3] for substring?
- 1 kudos
- 4877 Views
- 4 replies
- 0 kudos
CDC with Delta Live Tables, with AutoLoader, isn't applying 'deletes'
Hey there Community!! I'm using dlt.apply_changes in my DLT job as follows:dlt.apply_changes( target = "employee_silver", source = "employee_bronze_clean_v", keys = ["EMPLOYEE_ID"], sequence_by = col("last_updated"), apply_as_deletes = expr("Op ...
- 4877 Views
- 4 replies
- 0 kudos
- 4716 Views
- 1 replies
- 1 kudos
SQL connector from databricks-sql-connector takes too much time to convert to pandas
Hello,I am using querying my Delta Lake with SQL Connect and later want to explore the result in pandas.with connection.cursor() as cursor: cur = cursor.execute(""" SELECT DISTINCT sample_timestamp, value, name FROM de...
- 4716 Views
- 1 replies
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
